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Cross Reference to Related Applications 

5 [0001] This application is a continuation-in-part claiming priority under 35 U.S.C. § 120 
of U.S. Patent Application No. 10/207,677 filed on July 27, 2002, and entitled "MAN- 
MACHINE INTERFACE USING DEFORMABLE DEVICE." This application is herein 
incorporated by reference in its entirety for all purposes. 

Background of the Invention 

10 1. Field of the Invention 

[0002] This invention relates generally to alternative input mechanisms to computer 
systems, and more particularly to visually tracking a light, wherein a change in the light 
triggers an action on the part of the computer system. 

2. Description of the Related Art 

15 [0003] There has been a great deal of interest in searching for alternatives to input 
devices for computing systems. The keyboard and mouse for the desktop computing 
system are taken for granted at this time. However, for interactive entertainment 
applications in a "living room" environment, the keyboard and the mouse have failed to 
gain widespread acceptance. 

20 [0004] Tracking of moving objects using digital video cameras and processing the video 
images for producing various displays has been attempted also. However, these systems 
tend to rely on having a plurality of video cameras available for developing positional 
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information about the object based on triangulation. The cost for these systems becomes 
prohibitive for their introduction into the "living room" environment. 
[0005] However, in spite of the above knowledge and techniques, problems continue to 
hinder successful object tracking, and a particularly difficult problem is extracting 
5 precisely only those pixels of a video image which correspond unambiguously to an 
object of interest. For example, although movement of an object having one color against 
a solid background of another color, where the object and background colors vary 
distinctly from one another, can be accomplished with relative ease, tracking of objects, 
even if brightly colored, is not so easy in the case of multi-colored or non-static 

10 backgrounds. Changes in lighting also dramatically affect the apparent color of the object 
as seen by the video camera, and thus object tracking methods which rely on detecting a 
particular colored object are highly susceptible to error or require constant re-calibration 
as lighting conditions change. The typical home use environment for video game 
programs demands much greater flexibility and robustness than possible with 

15 conventional object tracking computer vision systems. 

[0006] Thus, an alternative input device must be able to be tracked under the home use 
environment by a single relatively inexpensive camera in order to become widely 
accepted. Additionally, the alternative input device must be convenient to use. While a 
glove worn on the hand of a user, where the glove includes sensors that are tracked by a 

20 camera to capture input, has been trialed, users have not embraced the glove. One of the 
reasons for the lack of enthusiasm for a glove is the inconvenience of having to 
continually remove and put on the glove. Furthermore, the alternative input devices 
described above do not easily adapt themselves to being able to indicate a mode change, 
functionally similar to a mouse click. 
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[0007] Thus, there is a need to solve the problems of the prior art to provide an input 
device capable of being tracked by a single video camera, wherein the input device is 
convenient for the user and capable of indicating a mode change. 
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Summary of the Invention 

[0008] Broadly speaking, the present invention fills these needs by providing a method 
and system that provides a device capable of producing one or more colors of light (or 
variations of a single or multiple light shades) where a change or variation triggers a 
5 mode change to the system controlling a display of image data. As used herein, a mode 
change refers to a discrete event or action triggered through the detection of a change in 
light emitted from an input device. It should be appreciated that the present invention can 
be implemented in numerous ways, including as a process, a system, or a device. Several 
inventive embodiments of the present invention are described below. 

10 [0009] In one embodiment a method for triggering input commands of a program run on 
a computing system is provided. The method initiates with monitoring a field of view in 
front of an image capture device. Then, a light source within the field of view is. 
identified. Next, a change in the light emitted from the light source is detected. In 
response to detecting the change, an input command is triggered at the program run on the 

15 computing system. 

[0010] In another embodiment, a method for detecting input commands from an input 
source within a field of sight of an image capture device is provided. The method 
initiates with minimizing an amount of light entering the image capture device. Then, a 
first color light signal is detected from the input source through the image capture device. 

20 Next, a change from the first color light signal to a second color light signal is detected. 
Then, a mode change is triggered in response to the change in the first color light signal. 
[0011] In yet another embodiment, a computer readable medium having program 
instructions for triggering input commands of a program run on a computing system is 
provided. The computer readable medium includes program instructions for monitoring a 
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field of view in front of an image capture device. Program instructions for identifying a 
light source within the field of view and program instructions for detecting a change in 
light emitted from the light source are included. Program instructions for triggering an 
input command at the program run on the computing system in response to detecting the 
5 change are included. 

[0012] In still yet another embodiment, a computer readable medium having program 
instructions for detecting input commands from an input source within a field of sight of 
an image capture device is provided. The computer readable medium includes program 
instructions for minimizing an amount of light entering the image capture device and 

10 program instructions for detecting a first color light signal from the input source through 
the image capture device. Program instructions for detecting a change from the first color 
light signal to a second color light signal and program instructions for triggering a mode 
change in response to the change in the first color light signal are provided. 
[0013] In another embodiment, a computing system having an input detection system, 

15 that determines when to trigger input commands of a main program run through the 
computing system is provided. The computing system includes an image capture device. 
Logic for monitoring a field of view associated with the image capture device and logic 
for tracking a position of a light source associated with an input object are included. 
Logic for detecting a color change in the light source and logic for triggering an input 

20 command at the main program run through the computing system, where the triggering is 
a result of the detected color change in the light source are included. 

[0014] In yet another embodiment, an input device for interfacing with a computing 
device is provided. The input device includes a body configured to be held within a 
human hand. The input device includes a light emitting diode (LED) affixed to the body 
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and a power supply for the LED. A mode change activator is integrated into the body, 
where the mode change activator is configured to cause a change of a color of a light 
originating from the LED. The color change is capable of being detected to cause a mode 
change at the computing device. 
5 [0015] Other aspects and advantages of the invention will become apparent from the 
following detailed description, taken in conjunction with the accompanying drawings, 
illustrating by way of example the principles of the invention. 
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Brief Description of the Drawings 



[0016] The invention, together with further advantages thereof, may best be understood 
by reference to the following description taken in conjunction with the accompanying 
drawings. 

5 [0017] Figure 1A is a simplified schematic diagram of a system having the capability of 
input detection from a light source in order to trigger a mode change in accordance with 
one embodiment of the invention. 

[0018] Figure IB is an alternative representation of the system depicted in Figure 1 A. 
[0019] Figure 2 is a simplified schematic diagram illustrating the capture of light from a 
10 light source through an image capture device in accordance with one embodiment of the 
invention. 

[0020] Figure 3 is a schematic diagram illustrating the determination of the location of a 
light source and the subsequent translation of that location to control movement of a 
corresponding cursor on a display screen in accordance with one embodiment of the 
15 invention. 

[0021] Figure 4 is a schematic diagram illustrating a scheme for enhancing a tracking and 
translation methodology in accordance with one embodiment of the invention. 
[0022] Figure 5 is a simplified schematic diagram illustrating a scheme for setting a scale 
dependent upon the distance a user is from the image capture device in accordance with 
20 one embodiment of the invention. 

[0023] Figure 6 represents alternative configurations of an input device in accordance 
with one embodiment of the invention. 

[0024] Figure 7 illustrates an alternative embodiment for the input devices illustrated in 
Figure 6. 
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[0025] Figure 8A is a simplified schematic diagram of a pair of input devices configured 
to communicate with a computing device in communication with a display monitor in 
accordance with one embodiment of the invention. 

[0026] Figure 8B is a schematic diagram of an alternative light input device configuration 
5 to the rings of Figure 7. 

[0027] Figure 9 is a flow chart diagram illustrating the method operations for triggering 
input commands for a program running on a computing system in accordance with one 
embodiment of the invention. 

[0028] Figure 10 is a flow chart diagram illustrating the method operations for detecting 
10 input commands from an input source within a field of sight of an image capture device 
in accordance with one embodiment of the invention. 
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Detailed Description of the Preferred Embodiments 



[0029] An invention is disclosed for an input device that is capable of emitting multiple 
colors/frequencies of light in order to trigger an event. In the following description, 
numerous specific details are set forth in order to provide a thorough understanding of the 
5 present invention. It will be apparent, however, to one skilled in the art that the present 
invention may be practiced without some or all of these specific details. In other 
instances, well known process steps have not been described in detail in order not to 
unnecessarily obscure the present invention. 

[0030] The embodiments of the present invention provide a user input device that is 
10 capable of emitting multiple colors of light which are captured through an image capture 
device. The changing from one light color to another initiates an event or action that can 
be displayed on a monitor. For example, the change from one light color to another can 
signify a mouse click, i.e., a mode change, to open a file or window for a personal 
computer, grab, drag or manipulate an image for a computer game, or even start a smart 
15 appliance. It should be appreciated that the input device includes some type of button, or 
actuator, which is used to change between light colors being emitted from a light emitting 
diode (LED). Thus, the input device will include some sort of power supply for the LED 
also. As will be explained by the embodiments described herein the user input device 
allows for the introduction of an experience that can be related to a mouse click while 
20 being convenient for the user. It should be appreciated that the light device is a "dumb" 
device. That is, no active communication takes place between the input device and the 
computer as the computer, in essence, watches for the light input device. In one 
embodiment, the light input device is configured as a flashlight. Of course, the light input 
device may be cordless, e.g., battery powered for the generation of light. 
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[0031] Figure 1A is a simplified schematic diagram of a system having the capability of 
input detection from a light source in order to trigger a mode change in accordance with 
one embodiment of the invention. Image capture device 104 is in communication with 
computing device 106, which in turn is in communication with display screen 108. Input 
5 device 100 includes a light source 102. For example, light source 102 may be a suitable 
light emitting diode (LED). Light emanating from LED 102 is detected by image capture 
device 104. It should be appreciated that image capture device 104 may be any suitable 
image capture device capable of detecting light from LED 102. For example, any suitable 
web cam or other camera may be used as image capture device 104. The location in 

10 space of LED 102 is used to control movement of cursor 1 10 on display screen 108 in one 
embodiment. That is, as input device 100, and associated LED 102, is moved in space, 
that movement is translated in order to move cursor 110. Additionally, as will be 
explained in more detail below, input device 100, through LED 102, may cause a mode 
change, similar to the mode change initiated by clicking a mouse for a desktop computer. 

15 As mentioned above, a mode change refers to a discrete event or action triggered through 
the detection of a change in light emitted from an input device. In essence, input device 
100 may be used as a computer mouse through color changes associated with LED 102. 
Exemplary mode changes include mode changes associated with mouse-type operations 
where the movement features of the mouse in conjunction with the selection features, i.e., 

20 clicking, double clicking, and right clicking, are incorporated into the system described 
below. That is, the tracking features of the system capture the movement of the input 
device while the light color changes provide the means to capture selection features. 
While a mode change has been described in relation to mouse-type operations, it should 
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be appreciated that the embodiments are not limited to mouse-type operations. That is, 
any suitable discrete operation or action is included here. 

[0032] Figure IB is an alternative representation of the system depicted in Figure 1A. 
Here, user 112 is holding input device 100 within his hand. LED 102 which is affixed to 
5 an end of input device 100, is within a detection region of image capture device 104. 
Image capture device 104 is in communication with computing device 106 which is in 
communication with display screen 108. In one embodiment, computing device 106 is 
any suitable video game console, e.g., the PLAYSTATION 2 console. Controller 116 is 
also in communication with computing device 106. Thus, user 112 may move input 

10 device 100 from an initial location, thereby changing the position of LED 102 relative to 
camera 104. This relative movement is then translated in order to move a cursor on 
display screen 108. Additionally, a mode change associated with the cursor being moved 
on display screen 108 is triggered through a change in the color of light emitted from 
LED 102. It should be appreciated that the embodiments described herein provide for a 

15 mouse-like device to be introduced in the living room for interactive entertainment and 
any other suitable application. 

[0033] Figure 2 is a simplified schematic diagram illustrating the capture of light from a 
light source through an image capture device in accordance with one embodiment of the 
invention. Here, user 112 is holding input device 100 which includes a light source 
20 thereon. Image capture device 104 monitors a field of view 118 through which light from 
the light source of input device 100 is detected. The light source associated with input 
device 100 is within plane 117, which corresponds to digitized screen 120. Here, an 
image of the light source associated with input device 100 is illustrated by region 122 of 
screen 120. It should be appreciated that the resolution of screen 120 may be associated 
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with any suitable resolution typical of a web cam or other suitable camera. In one 
embodiment, screen 120 is defined by a screen size of 320x240. Thus, as user 112 moves 
input device 100, the associated movement is captured through capture device 104 to 
determine a location of the light source within screen 120. It should be appreciated that 
5 the screen size and the imaging device size are decoupled. However, the screen and 
image device size are mapped in order to determine corresponding positions between the 
two. In one embodiment, the image device is mapped to a region of the screen. Here, 
most of the screen is used for displaying a scene, game image, etc., and there is a 
relatively small input palette in a corner or some other suitable region of the screen. 

10 [0034] Figure 3 is a schematic diagram illustrating the determination of the location of a 
light source and the subsequent translation of that location to control movement of a 
corresponding cursor on a display screen in accordance with one embodiment of the 
invention. Here, screen 120 defines an image of a light source as region 122. Region 122 
includes portions of pixel P a , Pt>, P c , Pd> P e , and Pf. The remainder of each of the pixels in 

15 screen 120, i.e., all pixels except pixels P a -Pf, are black. In one embodiment, ensuring 
that the remainder of the pixels is black is achieved through a masking operation. The 
masking operation includes reducing the size of an aperture of image capture device 104 
in order to minimize an amount of light allowed into the image capture device. In one 
embodiment, the aperture size may be adjusted electronically by adjusting the sensor 

20 game and exposure time. This scheme enhances the ability to detect a light source while 
reducing interference effects associated with background lighting. It should be 
appreciated that since the characteristics of the light input device and the image capture 
device are known, then the image capture device parameters (white balance, gain, 
exposure, saturation, etc.) may be set explicitly to track a particular pre-determined pixel 
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value, i.e., no calibration is required. As the input device is a light, the room lighting is 
not a factor here. Thus, an active method for detecting a light change is provided. 
[0035] Still referring to Figure 3, center 124 of region 122 is calculated through a 
centroid calculation in which the center's of each of pixels P a -Pf are calculated and then 
5 weighted according to the associated pixel value in order to determine the coordinates of 
center 124. The coordinates of center 124 are then mapped to display screen 128, which 
corresponds to the display screen being viewed by the user. Thus, movement of the light 
source will cause movement of region 122 on grid 120, which may also be referred to as a 
screen associated with the image capture device. The corresponding movement of region 

10 122 will be associated with the calculation of a new center. The new center will then be 
mapped to a location on screen 128 in order to move cursor 130 on screen 128 so that the 
user is given a feeling of control, over the movement of cursor 130 through the LED input 
device. As will be explained in more detail below, the input device may have a button or 
some other suitable activation device which when pressed will cause the respective LED 

15 to change to a different color from a previous color. This different color is then captured 
by image capture device 104. The detection of the different color results in different pixel 
values being associated with the change in color. For example, the pixels corresponding 
to region 122 will be associated with different values by the color change. The different 
pixel values will then signal the mode change similar to a mode change signal associated 

20 with a mouse click. Thus, a user may click and drag, highlight, etc., images on the 
display screen. That is, the user may perform any functionality achieved through a mouse 
associated with a computer. 

[0036] In one embodiment, the centroid calculation is performed as described hereafter. 
The pixels not associated with pixels P a -Pf, are assigned a value of 0 as no light is 
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detected, i.e., the pixels are black. It should be appreciated that the masking technique 
described above may be used to ensure that the image capture device can lock in on a 
light emanating from an input device by reducing interference from background lighting. 
Each of pixels P a -Pf are assigned a value corresponding to the amount of area of the pixel 
5 intersecting with region 122. In one embodiment, where pixel values are assigned from 
0-255, 0 corresponding to no light, pixel P e is assigned the highest value while pixel P f is 
assigned the lowest value. For exemplary purposes the pixel values of pixels P a , Pb, P c > 
P d , P e , and P f are 121, 230, 80, 123, 240, and 10, respectively. Each of pixels P a -Pf is 
associated with a respective pixel center point. Each of the two dimensional coordinates 

10 of each of the pixel centers is multiplied by the value of the respective pixel. These 
weighted values for one of the two dimensional coordinates are then summed together. In 
one embodiment, the summation of the weighted values for each coordinate is then 
divided by the summation of the pixel values associated with region 122 in order to 
provide the coordinates for the center of region 124. This technique may be described 

15 mathematically as: 

(x,y)center= £ [[(x pix ei center] [value of pixel for x pixe i center] ]/£(pixel values) and 
2 [[(y P ixei center] [value of pixel for y P ixeicenter]]/2:(pixel values) 

20 [0037] Here, (x,y)center represent the two coordinates of center 124, xpixei center 
represents the x coordinate for each of pixels Pa -Pf, and ypixel center represents the y 
coordinate for each of pixels Pa -Pf. Thus, center 124 corresponds to a certain location of 
the image of the capture device. This position corresponds to a location on screen 128. 
With reference to video frames, (x,y)center may be calculated for each frame of the video 



SONYP030/MLG 



14 



Patent Application 



and the location of (x,y)center is used to set a position of cursor 130 on screen 128. In 
one embodiment, a resolution associated with grid 120 is less than the resolution 
associated with screen 128, thereby enabling smooth movement of cursor 130 across 
screen 128. It will be apparent to one skilled in the art that a non-weighted centroid may 
5 also be determined, especially if the background is not known, e.g., the background is not 
all black. Here, the location of the centroid may not be as accurate as when the 
background is known, however, the accuracy is still suitable for the embodiments 
described herein. In one embodiment, the non-weighted centroid is calculated when the 
user is in a dark room or with an infrared LED and camera. It will be apparent to one 

10 skilled in the art that while Figures 1A, IB, 2, and 3 refer to a cursor, the embodiments 
are not limited to use with a cursor. In essence any suitable indicator that provides 
feedback on the second location of the input device may be used. For example, effects 
like distortion, brightening, darkening, telescope windowing, etc. may be employed to 
provide feedback on the second location of the input device. 

15 [0038] Figure 4 is a schematic diagram illustrating a scheme for enhancing a tracking and 
translation methodology in accordance with one embodiment of the invention. It should 
be appreciated that where a light source is captured through image capture device 104 and 
subsequently located within screen 120, the corresponding region associated with the 
light source is contained within one pixel. Thus, the subsequent translation to a cursor 

20 may cause the cursor movement to appear jumpy, due to the quantization effect of the 
discrete sampling of the image capture device. In order to alleviate the jumpiness, image 
capture device may be defocused to blossom or expand the region associated with the 
light source. For example, region 132 represents an initial capture of a corresponding 
light source. As can be seen, region 132 is contained within one block of grid 120, which 
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represents a single pixel. In order to expand or blossom region 132 the image capture 
device is defocused where regions 134 and 136 represent different defocusing parameters. 
Thereafter, the centroid of the expanded region may be calculated as discussed above. In 
one embodiment, a diffuser is placed over the LED to defocus the light source. For 
5 example, the diffuser may be a piece of tape that causes the light to diffuse. 

[0039] Figure 5 is a simplified schematic diagram illustrating a scheme for setting a scale 
dependent upon the distance a user is from the image capture device in accordance with 
one embodiment of the invention. Here, user 102a is at a first distance associated with 
image plane 117a while user 102b is at a second distance corresponding to image plane 

10 117b. It is not necessary that image capture device 104 has depth capability, as the 
corresponding scales from images captured at image plane 117a and image plane 117b 
may be used to provide a relative degree of distance corresponding to respective image 
areas occupied by user 102a and user 102b. According to this relative degree of distance, 
the amount of movement for input device 100 to cause a corresponding movement of a 

15 cursor on display screen 108 may be adjusted. For example, if the user is closer to image 
capture device 104, then larger movements may be used to correspond to a movement of 
a cursor as compared to smaller movements when the user is at a farther distance. 
[0040] Figure 6 represents alternative configurations of an input device in accordance 
with one embodiment of the invention. Input device 100a includes LED 102a-l and LED 

20 102a-2 located at opposing ends of the input device. Button 103 is included on the body 
of input device so that a user may press the button in order to trigger a light change for a 
corresponding LED. More than one button may be incorporated into input device 100a in 
order to accommodate multiple LED's in one embodiment. Input device 100b includes a 
single LED 102b at an end of the input device. Here, LED 102b is capable of emanating 
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multiple colors as button 103 is pressed. Input device 100c illustrates multiple LED's 
located adjacent to each other. Here, LED 102c-l is adjacent to LED 102c-2. As an 
alternative to input device 100c, the input device may have a fork-type configuration 
where an LED is affixed to each of the ends of the prongs (tines) of the fork. Button 103 
5 may also be used to trigger one of the LED's to emit light while another is off. Button 
103 may also be referred to as a mode change activator. A mode change activator is 
broadly defined as any suitable mechanism that may be used to cause the LED to switch 
between colors of light and/or variations of colors of light. For example, the mode 
change activator may be a button, a switch, a rotary dial, etc. In addition, the LED's may 

10 be located elsewhere on the body of the input devices shown in Figure 6. For example, 
LED's may be incorporated on the sides of the input device. Alternatively, a line of 
LED's along the side of the input device may be provided. In another embodiment, a 
large LED at an end of the input device is provided, thereby enabling a capture device to 
detect a change in shape when the input device is tilted. That is, the input device may be 

15 configured to enable the capture device to detect a change in angle of the input device 
relative to the capture device. For example, a user may angle the input device up, down 
or to the side in order to cause a certain mode change or response to the particular angle 
change. One skilled in the art will appreciate that numerous other suitable configurations 
are possible for the input device besides the configurations illustrated in Figure 6. Thus, 

20 the embodiments described herein are not limited to the exemplary configurations of 
Figure 6. 

[0041] Figure 7 illustrates an alternative embodiment for the input devices illustrated in 
Figure 6. Here, input device 100 is configured as a remote control device which includes 
LED 102 and infrared capability indicated by light 140. Thus, the input device may be 
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incorporated into a suitable remote control commonly used for television sets. In one 
embodiment, an LED capable of toggling between at least three colors is provided. Here, 
a third color may be used to provide functionality corresponding to a "right click" on a 
computer mouse. 

5 [0042] Figure 8A is a simplified schematic diagram of a pair of input devices configured 
to communicate with a computing device in communication with a display monitor in 
accordance with one embodiment of the invention. External input devices 100-1 and 
100-2 are configured to fit over a finger or thumb of a user's hands 142a and 142b. As 
shown, each of input devices 100-1 and 100-2 is capable of emanating light that is 

10 detected by image capture device 104 which is in communication with computing device 
106. While image capture device 104 is shown outside of the frame of computing device 
106, it should be appreciated that the image capture device may be integrated into the 
computing device in one embodiment of the invention. Input devices 100-1 and 100-2 
transmit different color light signals in one embodiment. Computing device 106 is in 

15 communication with display monitor 108. Computing device 106 transmits digital data 
to display monitor 108 so that the digital data can be viewed. Display monitor 108 may 
display text 146a, menu 146b and/or graphics 146c. Of course, it should be noted that 
any suitable digital data may be displayed. In one embodiment, where computing device 
106 is a game console, display monitor 108 displays graphics or a user interface 

20 associated with a game being played. 

[0043] Each of input devices 100-1 and 100-2, of Figure 8A, is configured to generate 
different colors of light. In one embodiment, input devices 100-1 and 100-2 may emanate 
a common color of light and at least one different color of light. It should be appreciated 
that button 144 may be used to toggle between the different colors of light. The light may 



S ONYP030/MLG 



18 



Patent Application 



be emitted through a LED on the side of the ring band of input devices 100-1 and 100-2. 
Alternatively, input devices 100-1 and 100-2 may be configured as thimbles, where a 
LED is affixed to the base of the thimble. In another embodiment, a single input device is 
employed rather than multiple devices. It should be appreciated that input devices 100-1 
5 and 100-2 are configured to fit comfortably over a user's finger or thumb. Thus, different 
sizes can be made available based upon a user's age, gender, etc. Alternatively, the input 
devices may be made to be adjustable. That is, elastic straps, or even VELCRO straps, 
may be incorporated to secure the input device to the user's finger or thumb in one 
embodiment of the invention. 
10 [0044] Figure 8B is a schematic diagram of an alternative light input device configuration 
to the rings of Figure 7. Here, thimble body 100-3 includes button 144 which is 
configured to change the light or frequency of light being emitted through LED 151. Of 
course, more than on LED may be located on the base of thimble 100-3, as discussed with 
reference to Figure 6. 

15 [0045] One skilled in the art will appreciate that the image capture device and the 
computing device include logic capable of providing the functionality described herein. 
The logic may include software elements and/or hardware elements. For example, the 
software elements may include software code processed through a central processing unit. 
The hardware elements include logic gates and circuitry configured to achieve the 

20 functionality described herein. It will be apparent to one skilled in the art that the 
hardware elements, e.g., logic gates, may be synthesized to execute the functionality 
described herein. Of course, the system may include a combination of software and 
hardware elements interacting to provide the desired outcome also. 
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[0046] Figure 9 is a flow chart diagram illustrating the method operations for triggering 
input commands for a program running on a computing system in accordance with one 
embodiment of the invention. The method initiates with operation 150 where a field of 
view in front of an image capture device is monitored. Here, the field of view may 
5 capture a light source emanating from an input device having an LED capable of emitting 
multiple colors of light, as described with reference to Figures 1 A through 2. The method 
then advances to operation 152 where a light source within the field of view is identified. 
As described above, the light source emanates from the LED. The method then proceeds 
to operation 154 where a change in light emitted from the light source is detected. Here, 

10 a button may be pressed in order to change the color of light emitted from an LED which 
is then captured by the image capture device. That is, pixel values will change when the 
change in the color of light occurs. As used herein, the change in the color of light may 
refer to an actual color change, e.g. from red to green. However, the change of color may 
also refer to a variation of the color, e.g., one yellow variant to another yellow variant. In 

15 essence, any suitable change that will cause a corresponding change in the pixel values 
associated with the colors or variants may be used here. The method then moves to 
operation 156 where an input command at the program run on the computing system is 
triggered in response to the change in the color of light. For example, a mode change 
associated with a mouse click on a desktop computer may be triggered here. Thus, click 

20 and drag functionality, highlighting functionality, and any other suitable functionality 
achieved through a mouse click is capable of being introduced in the "living room" 
environment without the traditional mouse and keyboard hardware. In addition, the input 
command triggered by the light change can be an action, a movement cue, a modality 
change, etc. 
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[0047] Figure 10 is a flow chart diagram illustrating the method operations for detecting 
input commands from an input source within a field of sight of an image capture device 
in accordance with one embodiment of the invention. The method initiates with 
operation 160 where an amount of light entering the image capture device is minimized. 
5 That is, an aperture of the image capture device may be reduced in order to mask 
background lighting. Thus, the minimization of the background lighting will enhance the 
tracking capability of a light source such as the input device described herein. For 
example, the reduction in the amount of light entering the image capture device may be 
achieved as described with reference to Figure 3. The method then advances to operation 
10 162 where a first color light from the input source is detected through the image capture 
device as described with reference to Figures 2-4. Here, the first color light is detected 
and correlated to an image screen. 

[0048] The method of Figure 10 then proceeds to operation 164 where the first color light 
being changed to a second color light is detected. Here, the change in the color of light 

15 may be triggered through pressure applied to a button or mode change activator on the 
input device as described above. One skilled in the art will appreciate that the color 
change, or frequency change, may be detected by examining corresponding pixel values 
associated with an image capture device. The method then moves to operation 166 where 
a mode change is presented in response to the change in the first color light. The mode 

20 change may act to enable a user to perform click-and-drag operations, highlighting 
operations, or any other suitable operations associated with a mode change such as 
enabled through a computer mouse. Additionally, when the second color light is changed 
back to the first color light, the end of a click-and-drag operation or highlighting 
operation is signified. Alternatively, a third color light may be used to enable 
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functionality associated with a "right" mouse click. It should be noted that the mode 
change is displayed on a display screen similar to the mode change for a computer mouse 
in one embodiment of the invention. Additionally, one skilled in the art will appreciate 
that while Figure 10 is described with respect to color changes, the invention is not 
5 limited to color changes as variations of the same color of light may be used, e.g., shades 
of colors. Furthermore, different frequencies of light may be used instead of different 
colors. For example, infrared light may be used with a visible wavelength of light. As 
discussed above, any change to a light source that is capable of causing a change in pixel 
values associated with an image capture device may be used for the embodiments 

10 described herein. 

[0049] In summary, an input device capable of triggering a mode change through a multi- 
color, or multi-frequency, light generator is provided. Of course, multiple collocated 
lights having different colors or frequencies may be used to provided the same effect. 
Thus, any light generating structure can be used, whether it is a solid-state device such as 

15 a diode, or one or more conventional light bulbs. In one embodiment, the light is 
provided by one or more LEDs that can be coupled or installed onto a pen-like object. 
The light emitted from the input device is tracked by an image capture device. The image 
capture device can take on any number of forms, including a camera, an array of charged 
coupled devices (CCDs), a digital camera, a conventional camera that is coupled to a 

20 digitizer, or a webcam. In general, the image capture device should be able to detect light 
from the input device (or any object capable of emitting light). Preferably, the input 
device will generate enough light so as to enable the camera to detect a spot of light, 
which may be bright relative to other surrounding light. 
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[0050] Thus, in one embodiment, similar to a computer mouse input device, the light 
from the input device may be set to always be on when it is detected by the camera. 
Thereafter, to cause control, the user may press a button, or some other suitable triggering 
mechanism, on the input device to change a color of light (or a frequency) being emitted. 
5 The webcam captures the color/frequency change, thereby enabling the color change to 
act as a button press for a mouse, i.e., mouse click. It should be noted that any suitable 
number of colors may be tracked and that the input device may have multiple buttons for 
the multiple colors or the input device may have a single button that cycles through the 
multiple colors. In another embodiment, instead of changing visible color, the change 

10 may be from a first color that changes to general light, as is typical of standard flash 
lights. In still another embodiment, the input device can flicker between a first light and 
then to an off state, where not light is emitted at all. For example, an infrared LED may 
alternate between a first light and an off state. In such a case, when the light is off, the 
light may stay off for a set period of time. The timing of the off state can then cause one 

15 or more same, similar or different states or changes to occur on a display screen. 

[0051] Furthermore, the input device described herein enables enhanced functionality for 
interactive entertainment applications. For example, with respect to sports video games, 
a user may use the input device to design a play, e.g., a play for a football game, 
basketball game, etc. Strategy games, such as chess and war games, which were 

20 previously limited due to the difficulty in adopting the joystick controller to function as a 
mouse-type device become more user friendly when the input device described herein is 
used for mouse-type input, instead of the joystick. With respect to a video game that 
incorporates strategy, a user may be on a hillside directing armies to attack an enemy. In 
another embodiment, the input device may be used for television programming. The 
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ability to incorporate a mode change associated with the change of a light color in 
conjunction with the capability to track the movement of the light source enables the 
accomplishment of these features. 

[0052] The invention may employ various computer-implemented operations involving 
5 data stored in computer systems. These operations are those requiring physical 
manipulation of physical quantities. Usually, though not necessarily, these quantities take 
the form of electrical or magnetic signals capable of being stored, transferred, combined, 
compared, and otherwise manipulated. Further, the manipulations performed are often 
referred to in terms, such as producing, identifying, determining, or comparing. 

10 [0053] Any of the operations described herein that form part of the invention are useful 
machine operations. The invention also relates to a device or an apparatus for performing 
these operations. The apparatus may be specially constructed for the required purposes, 
or it may be a general purpose computer selectively activated or configured by a computer 
program stored in the computer. In particular, various general purpose machines may be 

15 used with computer programs written in accordance with the teachings herein, or it may 
be more convenient to construct a more specialized apparatus to perform the required 
operations. 

[0054] Although the foregoing invention has been described in some detail for purposes 
of clarity of understanding, it will be apparent that certain changes and modifications may 
20 be practiced. For example, although specific examples have been provided for use in 
relation to video gaming, the applications can be applied to any computer or computing 
device that will require some interaction. The computing device can be a single stand 
alone unit or can be interconnected to other computing devices over a local or global 
network, such as the Internet. Accordingly, the present embodiments are to be considered 
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as illustrative and not restrictive, and the invention is not to be limited to the details given 
herein, but may be modified within the scope and equivalents of the description. 

What is claimed is: 
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